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Abstract: Multiple choice questions (MCQs) and short essay questions (SEQs) are common methods of the 
assessment of medical students in courses of pharmacology. Poorly constructed test items (questions) are a 
widespread problem resulting in failure to assess learning objectives. It has been reported that there are 36.0% 
to 65.0% flawed test items in medical education assessment tools. Thus, the objective of this study was to 
evaluate MCQs by determining the item writing flaws (IWFs) and to evaluate the SEQs by determining the 
cognitive level of each item. Four pharmacology tests were administered to third year pharmacy students at 
Department of Pharmacology, Faculty of Pharmacy, Omar Al-Mukhtar University, Bayda, Libya. These were 
evaluated by determining the IWFs and the level of the cognitive domains. Based on Buckwalter’s 
modification of Bloom’s taxonomy cognitive level, for the SEQs, 30.0% of the questions were attempted to 
check recall of information, 26.0% were attempted to evaluate understanding and interpretation of data and 
43.0% of the questions were attempted to check the application of knowledge for solving a particular problem. 
For the MCQs, 94.6% of the questions were attempted to evaluate the understanding and interpretation of 
data. For the [WFs, there were more than 40.0% of flawed questions. The most common writing flaws were 
the negative stem (47.4%), unfocused item (16.0%), non-homogenous in grammar and contents (10.0%), all 
the above (10.0%) and clang association (05.0%). In a short essay, the SEQs were of excellent quality because 
they were equally distributed among the three levels of cognitive (level I, II and II). On the other hand, the 
most common mistakes IWFs of the MCQs were the negative stem (47.0%) and the idea was not clearly and 
concisely stated in the stem (16.0%). This study concludes that questions in SEQs are valid to measure the 
learning objective but MCQs were not in pharmacology courses in Libya. 


Introduction 


The main objectives of any educational program 
are to enable learners to develop different cognitive 
abilities, such as recalling the fundamentals or the 
principles (lower cognitive order) or problem 
solving or clinical reasoning. To ensure that the 


learners acquired the intended performance or 
competency, evaluation or assessment must be 
conducted and aligned with the learning objective, 
during or at the end of the education program. In 
pharmacology education, assessment(s) is an 
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overly critical tool that evaluates the learners’ 
knowledge, skills and attitude consequently 
learners’ outcomes or educational quality. 
Different type of assessment demands more 
emphasis on the certain construct (category) for 
example objective structured clinical examination 
(OSCE) use standardized patients to evaluate 
counseling, clinical procedure, application and 
clinical problem-solving. The OSCE, for instance 
in Pharmacy School, must be passed by 
examination to be a practice pharmacist. On the 
other hand, MCQs and SEQs are used to assess 
basic knowledge and clinical reasoning based on 
learning objectives resulting either in passing or 
failing in educational courses [1]. In any 
assessment, two elements: cognitive level and item 
writing principles are essential in governing the 
quality and validity of the assessment. 


Bloom specifies cognitive abilities into knowledge 
(facts, basics and principles), comprehension, 
application, analysis, synthesis and evaluation. 
Most educational programs such as medical and 
pharmacy practice attempt to develop a medical 
curriculum (both instruction and assessment) that 
incorporates not only base information (basic and 
principles) but also complex cognitive abilities [2]. 
Thus, quality crafted assessment should reflect the 
basic and the more complex skills. Students are 
evaluated based on these cognitive abilities [2]. 
According to the revised Bloom’s taxonomy, most 
of the learning objectives and the examination 
questions were classified under the lower level of 
cognitive domains [3]. Despite the importance of 
complex cognitive skills, most educational 
program measure just the recall or basic or 
fundamental information [2]. Measuring just basic 
knowledge results in learners with poor learning 
outcomes. Recently, modified Bloom’s cognitive 
levels categorize the cognitive levels into level I 
(recall of information), level II (understanding and 
application) and level II (problem-solving). In 
addition, for writing MCQs guidelines, there was 
no recognized guidelines for writing well- 
constructed MCQs. Nevertheless, Haladyna and 
Downing [4] developed guideline for writing well- 
constructed MCQs. Following the standard items 
of writing principles results in well-crafted 


questions. However, failing or violating the 
guidelines or principles of writing MCQs results 
high frequency of flawed items in the tests studied 
(item writing flaws, IWKs) [5], pausing the 
challenge to learners. Further, flawed items in 
MCQs lead to incorrectly classifying students as 
failed when they should have been classified as 
passed. MCQs consist of the stem (posing 
question), 3 - 5 alternative options: one correct 
answer (key) and distractors (2 - 4 wrong answers). 
With the ease of scoring, higher reliability and 
validity, MCQs is commonly used either alone or 
in combination to measure the basic knowledge or 
synthesis, application and problem-solving skills 
[6] as described by the learning objectives. On the 
other hand, essay-type assessment is the time- 
consuming but can be used alone or in combination 
to measure the ability of learners to recall facts and 
to apply or solve problems [6]. One of the most 
common problems of MCQs is the IWFs, i.e., 
unfocused stem or implausible distractors. Also, 
the deficiency of the SEQs is the focus on the low 
order of cognitive domain, such as the ability to 
recall facts. Flaw items or low order cognitive 
questions fail to assess learning objectives [6]. In 
short, the SEQs may emphasize one level of the 
modified Bloom’s cognitive levels while the 
MCQs may have flawed questions such as 
negatively worded items or unfocused items. Thus, 
the purpose of this study was to evaluate MCQs and 
SEQs items in undergraduate students of the 
Pharmacology course by determining IWFs in 
MCQs and cognitive levels of each item in MCQs 
and SEQs in Libya. 


Materials and methods 


This analytical study was conducted at Department 
of Pharmacology, Faculty of Pharmacy, Omar AI- 
Mukhtar University, Bayda, Libya. The study was 
conducted between the academic years (2020 and 
2022). All the examinations were performed by 
qualifying lecturers of pharmacology with 
experience in teaching for more than five years. 
Different methods of assessments such as MCQs 
and SEQs were used as modules of assessments. 
The Department developed all the assessment 
questions in multiple-choice or in short essay 
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questions. Four pharmacology examinations 
involving MCQs and SEQs according to Tables 1 
and 2. Each question in the MCQs was analysed to 
evaluate for IWFs (Table 2). In addition, MCQs 
and SEQs were analysed to evaluate the cognitive 
level (Table 1). The cognitive levels of the 
assessment tools were analyzed using the 
modification of Bloom’s taxonomy [7]: Level I 
includes questions which attempt to check recall of 
information. Level II includes questions that 
attempt to evaluate the understanding and 
interpretation of the data. Level UI includes 


questions that attempt to evaluate the application of 
knowledge for solving a particular problem (Table 
1). For determining types of IWFs standard criteria 
given by previous studies [4, 5, 8] were used and 
commonly occurring violations of item-writing 
guidelines were identified (Table 2). Although 
there were no humans involved in the descriptive 
study, we mean the study was assessing the validity 
of the pharmacology examinations. Ethical consent 
was obtained from the Department of 
Pharmacology, Faculty of Pharmacy, Omar Al- 
Mukhtar University, to evaluate the examinations. 


Table 1: Modified Bloom’s taxonomy 


Level I Knowledge 
Level If Comprehension and application 


Level IIT Problem solving 


Recall of information 
Understanding and interpretation of data 


Use of knowledge and understanding in new circumstances 


Table 2: Judging the MCQs according to the presence of IWFs 


Type of IWFs 


Negative stem 


Idea is not clearly and concisely stated in the stem (unfocused) 


Clue to the right answer (clang association) 


Not homogenous in grammar or in content structure 


Using all above 


Using none of the above 


Data analysis: This is a quantitative study and a 
descriptive statistic has been used to analysis the 
percentage of the cognitive level in MCQs and in 
SEQs, and the percentage of flawed items in the 
four tests. Based on Haladyna and Downing which 
have developed guidelines for writing quality 
crafted questions [9], therefore, a prediction can be 
obtained if MCQs or SEQs are well constructed 
and can measure the learning outcomes. 


Results 


In this study, the examinations are designed to 
assess lower cognitive skills such as recall of 
information and higher cognitive abilities such as 
problem-solving quality. Items in the MCQs or in 
the SEQs are designed to test the student's ability 
to recall, understand, to apply what the students 
have learned. To assess students’ academic 


successes, these assessments should not have any 
IWFs and the questions should be evenly 
distributed between the different cognitive levels. 
To verify if the items in MCQs or SEQs, assess the 
student’s ability in lower and upper cognitive skills 
and free of any IWFs, four pharmacology 
examinations were selected and analyzed for the 
level of cognitive ability and the occurrence of 
writing flaws. For the IWFs flaws, (Figures 1 and 
2) of the one hundred questions, there were more 
than 40.0% flawed questions. What are the most 
common writing flaws? The most common writing 
flaws were the negative stem (45.0%), unfocused 
item (15.0%), non-homogenous in grammar and in 
contents (10.0%), all the above (10.0%), clang 
association (05.0%) and none of the above 
(05.0%). For the level of cognitive ability, in SEQs, 
Figure 3, the questions were scattered testing the 
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recall of information (25.0%), application (20.0%) and interpretation. In short, in SEQs, there was a 
and problem-solving (40.0%). Nevertheless, in scattered distribution of the cognitive levels while 
MCQs, Figure 3, only 03.0% of the question were in MCQs, most of the MCQs were intended for the 
evaluating the recall of information, 02.0% were understanding and interpretation of the data 
evaluating the problem solving while many of the (94.6%). 


questions (94.6%) were testing the understanding 
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Figure 3: Percentage of different cognitive levels in four pharmacology tests 
Where Cl, C2 and C3 are represent the cognitive level1, II and III, respectively. 
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Discussion 


Assessment is an essential part in the learning 
process. It usually spreads to cover the lower 
cognitive abilities such as assessing learners to 
recall factual knowledge or assessing higher 
cognitive skills for instance learner's ability to 
analyse, apply or solve problems [6]. In addition, 
the occurrence of writing flaws or uneven 
distribution of the cognitive levels in writing the 
test reduces the test validity or quality in assessing 
the learners’ academic achievements. This study 
found that almost half of the flawed questions were 
in four different pharmacology tests in Libya. 
Poorly crafted test items are common problems 
resulting in the wasting of time, resources and 
money [10]. In medical education, 36.0% - 65.0% 
of test items were flawed and due to flawed MCQs, 
10.0% - 15.0% of the students who failed should 
have passed [6]. As a result, the flawed tests are not 
valid to measure the students learning [9]. Also, the 
most typical of writing flaws were negative stems 
(45.0%), unfocused stem (15.0%), non-harmonized 
grammar (10.0%), all of the above (10.0%) and 
none of the above (05.0%). Data from the current 
study positively correlate to the previous study in 
which found that most MCQs flaws are negatively- 
worded stems, unfocused-item stems, and use of 
the all of above and none of the above [6]. 


The multiple assessment formats, MCQs_ and 
SEQs, are used not only to assess the lower order 
cognitive skills (level I): recall of facts but also to 
assess the higher cognitive abilities (level ID: 
understanding and application (level III) and 
problem-solving [6, 11]. In MCQs, the current 
study shows that only 03.0% of the question item 
assessed recalling of facts, 02.0% of the questions 
assessed problem-solving and 94.6% of questions 
assessed the understanding and application. On the 
other hand, in SEQs, 30.0% of the questions were 
in level I (recall of facts), 26.0% were in 
understanding and application (level II) and 43.0% 
were in the problem-solving (level III). Present 
findings indicated that questions in MCQs and 
SEQs, using Bloom’s taxonomy cognitive levels 
were unevenly distributed between cognitive 


levels. It is a potential that the questions measure 
the entire span of Bloom’s cognitive level. The 
current study also showed that a greater proportion 
of SEQs (30.0%) than MCQs (03.0%) were testing 
lower level of cognitive skills. Similarly, greater 
SEQs (26.0% level Il and 43.0% level III) than 
MCQs were evaluating the higher cognitive skills. 
However, SEQs were evenly distributed between 
cognitive levels. In correlation with the literature, 
Palmer et al. [11] showed that a greater proportion 
of questions tested lower levels of cognitive skills. 
This means that the assessment (i.e., MCQs) did 
not cover the spectrum of Bloom’s cognitive level 
resulting in inferior quality or validity assessment. 
The present study showed that a greater proportion 
of SEQs (30.0%) were testing lower levels of 
cognitive skills. In line with the literature, Palmer 
et al. [11] showed that more proportion of questions 
testing lower level of cognitive skills, what does 
this mean? This intended that the assessment (i.e., 
MCQs) did not cover the spectrum of Bloom’s 
cognitive level resulting in inferior quality or 
validity assessment. Further, it is a trend to find 
most of the questions focus on testing low-order 
thinking skills such as what are the side effects of 
propranolol?, Or which of the following 
medications are used for pregnancy hypertension?. 
These types of questions assess students on 
recalling or remembering facts, they are easy to 
design. In contrast, it is not common to find 
questions that assess higher order thinking skills, 
for instance, compare the effects of high-dose 
(1000 mg/d) and low-dose (100 mg/d) aspirin on a 
long-term cardiovascular patient?. 


Conclusion: This study indicates that questions in 
SEQs are evenly distributed between cognitive 
levels while in MCQs, the questions are unevenly 
distributed in Libyan medical universities. Also, 
the assessment has 50.0% flawed item and most 
item writing flaws are negatively worded stem, 
unfocused stem, non-homogeneous in grammar, 
using all the above and none of the above. Thus, it 
can be concluded that SEQs are valid to measure 
the learning objective but MCQs are not in Libyan 
medical universities. 
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